F-method for continuous restriction operators 

Dedicated to Professor Michael Eastwood for his 60th birthday 

Toshiyuki KOBAYASHI 

Kavli IPMU, and Graduate School of Mathematical Sciences 

The University of Tokyo 
E-mail address: toshi@ms.u-tokyo.ac.jp 

Abstract 

We prove some finiteness theorems for continuous (respectively, 
differential) operators that intertwine two induced representations of 
a reductive Lie group and a reductive subgroup. We then extend 
the '.F-method' from local to non-local operators, aiming for detailed 
analysis of such operators. We illustrate this general idea by concrete 
examples in conformal geometry, and explain how we can discover 
functional equations among such operators by the F-method. We 
also indicate how continuous operators with meromorphic parameters 
yield conformally equivariant differential operators as residues. 
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1 Finiteness theorems of restriction maps 

Let G be a real reductive linear Lie group, and P a parabolic subgroup. 
Associated to a finite dimensional representation A : P — > GLc(V), we define 
a homogeneous vector bundle V := G x P V over the real flag variety X := 
G/P. Then the group G acts continuously on the space C°°(X, V) of smooth 
sections, which is endowed with the Frechet topology of uniform convergence 
of finite derivatives. 

Suppose G' is an (algebraic) reductive subgroup of G, P' is a parabolic 
subgroup of G', and Y := G'/P'. For a finite dimensional representation 
v : P' — > GLc(W), we define similarly a homogeneous vector bundle W := 
G' Xpi W over Y, and form a continuous representation of G 1 on C°°(Y, W). 
We denote by Hom G /(C 00 (G'/P, V), C°°(G'/P', W)) the space of continuous 
G'-homomorphisms. 

Assume further that 



Then we have a natural G'-equivariant morphism t : y — >■ X. With this 
morphism l, we can define a continuous linear operator T : C°°(X, V) — >■ 
C f0O (Y, W) to be a differential operator in a wider sense than usual by the 
following local property: 



In the case Y = X, this definition is equivalent to that T is a differential oper- 
ator in the classical sense by Peetre's theorem. Let Diff G /(C°°(G'/P, V), C°°(G'/P', W)) 
denote the space of G'-intertwining differential operators. 
The object of our study is the following two spaces: 



(1.2) Diffc^C 00 ^, V), C°°(Y, W)) C Hom G ,(G°°(A, V), C°°(y, W)). 



(1.1) 



F'cPn G'. 



Supp(T/) C t-^Supp /) for any / G G°°(X, V). 



Our project is: 
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Project A. Construct explicitly differential (local)/ continuous (non-local) 
intertwining operators, and classify them. 

B. Single out appropriate settings in which the first problem makes sense. 
The recent work || 18, |25], |26| is a part of Project A, whereas the 



general theory of restrictions of representations [TPS], [T5], [IT], Q contributes to 
Project B. In fact this project reflects some special aspect of the branching 
problem. Finding explicit formulae of operators would make sense only if such 
operators are natural ones. Thus, in Project B, we wish to single out nice 
settings that assure the spaces in (|1.2|) to be finite dimensional, of uniformly 
bounded dimensions, or even preferably, one-dimensional. For this, we can 
apply the general theory of restrictions of representations with respect to 
reductive groups G D G' . The rest of this section will be devoted to an 
up-to-date summary of Project B. We shall discuss Project A in Sections 
to|. 

Let us begin Project B with the most general case, namely, the case 
where P and P' are minimal parabolic subgroups of G and G', respectively. 
Then is automatically satisfied (with P replaced by its conjugate, if 

necessary). We state two theorems on the upper estimate of the dimensions 
in this case: 

Theorem 1.1 (local operators). Assume rank]g G = rank^ G' . Then 

(1.3) dimDiff G ,(C~(X,V),C' 00 (y, W)) < oo 

for all finite dimensional representations V of P and W of P' . 
Remark 1.2. We can relax the real rank assumption of Theorem |1.1| by 

(1.4) 0M S ' hyp n ^{o}, 

see (11.71) below. 



Theorem 1.3 (non-local operators, [0]). 1) (finite multiplicity) The fol- 
lowing two conditions on the pair (G, G') are equivalent: 

(i) For all finite dimensional representations V of P and W of P', 
(1.5) dimRom G/ {C 00 {G/P,V),C 00 {G'/P',W)) < oo. 

(ii) There exists an open P' -orbit on G/P. 
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2) (uniformly bounded multiplicity) Let G be a simple Lie group. We write 
g and g' for the complexifications of the Lie algebras g^ and g' R of G and G' , 
respectively. Then the following condition on the pair (G, G') is equivalent: 



(i) 



(1.6) supsupdimHom G ,(C 00 (X,V),C 00 (r, W)) < oo, 

V w 

where the supremum is taken over all finite dimensional, irreducible 
representations of P and P' , respectively. 

The pair (q,q') is a strong Gelfand pair, i.e., the pair (g,g') is one 
of (sl(n + l,C),gl(n,C)), (sl(n + 1, C),s[(n, C)) (n ^ I), (so(n + 
l,C),so(n,C)), org = g'. 



Theorem |L3] (1) suggests that the orbit structure P'\G/P is crucial to 
understand HomG'(C°°(X, V), C°°(Y, W)). In fact, the orbit structure is used 
as a first step of the classification of this space (see J2fJ) in the setting that 
we discuss in Section |j. A remarkable feature is that the condition (ii) in 
Theorem |1.3| (2) depends only on the complexification (g, g') whereas the 
condition (ii) in Theorem |L3] (1) depends on the real form (g K , g^). 

Comparing Theorems |TTT| and |1.3|, we see that the space of continuous 
intertwining operators (i.e. the right-hand side of ( |1.2|) ) may be much larger 
than that of differential ones (i.e. the left-hand side of (|1.2|) ). Namely, 
'appropriate settings' in Project B should be different for local and non-local 
operators: 



Example 1.4. Let (G, G') = (GL{A,R),GL(2,R) x GL(2,R)). Then ([TJ 
holds, whereas (OI) fails. 



Example 1.5. Let (G, G') = (H x H.dmgH) with H = 0(p,q). Then the 
condition (ii) of Theorem |1.3| (1) holds if and only if min(p, q) < 1. Again, 



T73| ) holds but (|1.5| ) fails if min(p, q) > 2. Continuous intertwining operators 



were studied in |j] in the case q = 1. 



The proof of Theorem |1.3| is based on the compactification of the group 
manifold with normal-crossing boundaries, and on the theory of microfunc- 
tions and hyperfunctions developed by M. Sato, M. Kashiwara, T. Kawai, 



and T. Oshima []T2"| |. We note that the proof [TM| does not use any case-by-case 
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argument. Alternatively, the 'only if part of Theorem |1.3| (2) could be de- 
rived from a recent multiplicity-free theorem J^J and from the classification 
of real forms of strong Gelfand pairs. 

We give a proof of Theorem [O] in a more general form below. Let us fix 
some notation. A semisimple element H of a complex reductive Lie algebra 
is said to be hyperbolic if the eigenvalues of ad(if) e Endc(fl) are all real. 
Given a hyperbolic element H, we define subalgebras of g by 

n+ = n + (if), I = [(if), xC = xC(H) 

to be the sum of the eigenspaces with positive, zero, and negative eigenvalues, 
respectively. Then p(ff) := 1(H) + n + (if) is a parabolic subalgebra of 0. Let 
0k be the Lie algebra of a real reductive Lie group G, and := jjr ®r C. If 
if is a hyperbolic element of 0k, then 

P(H) :={geG: Ad(g)p(H) = p(ff)} = L(H) exp(n+ (if)) 

is a parabolic subgroup of G, and pm.(H) := p(H) fl 0k is its Lie algebra. 
We define the following subset of by 

(1.7) 0K ghyP := {H ^ 0r : ad (if) is hyperbolic, and L(H) is amenable}. 

Then pu(H) is a minimal parabolic subalgebra of 0k for all H e 0K g ' hyP - 

Definition 1.6. We say a parabolic subalgebra p of is ^-compatible if 
there exists a hyperbolic element if in 0' such that p = p(H). We say P is 
G'-compatible if we can take H in 0^. 

If P is G'-compatible, then p' := p PI0 7 becomes a parabolic subalgebra of 
; with Levi decomposition p' = (tflfl') + (n + n fl') and P' := P(~)G' becomes 
a parabolic subgroup of G'. 

By abuse of notation, we write C\ — > X for the line bundle associated 
to (A, V) if V is one dimensional. We write A ^> if (dX, a) 3> for all 
a G A(n + ,j) where j is a Cartan subalgebra of t. 

Theorem 1.7 (local operators). Let q' be a reductive subalgebra of q. Sup- 
pose p = I + n + is q' -compatible. 

1) (finite multiplicity) For any finite dimensional representation V and W 
of the parabolic subgroups P and P' , respectively, we have 

dimDiff G ,(C ,00 (A,V),C 00 (r,W)) < oo. 
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2) (uniformly bounded multiplicity) If (g, g') is a reductive symmetric pair 
and n + is abelian, then 

supdimDiff G; (C 00 (X,£ A ),C 00 (F, W)) = 1, 
w 

for any one- dimensional representation C\ of P with X ^> 0. Here W runs 
over all finite dimensional irreducible representations of P' . 

Proof. 1) We extend the classical duality between Verma modules and prin- 
cipal series representations (e.g. ||) to the context of the restriction for 
reductive groups G I G', and we have the following bijection: 

H Om(B ^ (E7(fl0<8^ 

see Corollary 2.9]. Here (X v ,V y ) denotes the contragredient represen- 
tation of (X,V). Then the proof of Theorem [O] is reduced to the next 
proposition. □ 

Proposition 1.8. Let g' be a reductive subalgebra of g. Suppose that p = 
[ + n + is g' -compatible. 

1) For any finite dimensional p-module F and p' -module F' , 

dimHoHy([/(0') ® u(pf) F',U(g) ® u{p) F) < oo. 

2) If {&, q') is a symmetric pair and n + is abelian, then 

supdimHony([/(g') ® u(v/) F',U(g) ® u{p) C A ) = 1 

F' 

for any one- dimensional representation C\ of p with X 0. Here the supre- 
mum is taken over all finite dimensional simple p' -modules F' . 

Proof. 1) The proof is parallel to []T7, Theorem 3.10] which treated the case 

where F and F' are simple modules of P and P', respectively. 

2) See [171, Theorem 5.1] for the proof. □ 

See also [15, Theorem B] for an analogous statement to Theorem |T7J (2) 
which was formulated in the context of unitary representations. 

Proof of Theorem [Q| . If if is a generic hyperbolic element H in g ; R , then 
Pr(H) is a minimal parabolic subalgebra of 0r owing to the rank assumption. 
Then Theorem ITTT1 follows from Theorem 11.71. □ 
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Remark 1.9. In most cases, for each representation (A, V) of P, there exist 
continuously many representations (z/, W) of P' such that 



Hom G ,(C 00 (X,V),C 00 (r, W)) ^ {0}. 

However, there exist a few cases where X = Y and G ^ G", and consequently, 
there exist only a finite number of representations (u, W) of P' such that 

Kom G >{Cr{X,V),C' 30 {Y,W)) ^ {0} 

([[U| Theorem 3.8]). Even in this case, Project A sheds some new light on 
the interaction of the classical analysis and representation theory (e.g. Weyl 
calculus), see §, [21 for (G, G') = (GL(2n, F), Sp(n, ¥)), F = R,C. 



2 F-method for continuous operators 

The 'P- method' is a powerful tool to find singular vectors in the Verma 
modules by using the algebraic Fourier transform [113. We developed the 'P- 
method' in the previous papers p3| , |25| , and constructed new covariant dif- 
ferential operators that generalize the classical Rankin-Cohen bi-differential 
operator || |7], 28] and the Juhl's conformally equivariant differential operator 
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In this section, we follow the notation of P5 |, and generalize the idea of 
the P-method from local to non-local operators. Concrete examples will be 
discussed in Section [|. 

We retain the setting of the previous section. Let Gc be a complexification 
of G. According to the Gelfand-Naimark decomposition q = n~ + t + n + of 
the complex reductive Lie algebra q, we have a diffeomorphism 

n"xL c xn + 4 G c , (A, £, Y) ^ (exp A)£(exp Y), 

onto an open subset containing the identity in a complexification Gc of 
G. Let 



P± . Gr c 



Po : G 1 ^ L 



be the projections characterized by the identity exp(p^(g))p (g) exp(p + (g)) 
g. We define the following maps a and 0. 



(2.1) (a,/3): 0f 



d 



(Po (■ 



JD„C\ 



e~e"),p- (e tD e c )) 



t=o 
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For D G g, we may regard (3(D, • ) as a holomorphic vector field on n via 
the identification n~ 3 C i-» f3(D, C) G n~ ~ T c n~ . 
Let N± := exp(n R ). Then 

(2.2) i: n R ->• G/P, Z i-> (exp Z)P 

defines an open Bruhat cell N~P/P in the real flag variety G/P. 

We denote by C 2p the one- dimensional representation of P on A dimn (g/p). 
The infinitesimal representation will be denoted by 2p. 

Let (A, V) be a finite dimensional representation of P with trivial ac- 
tion of N + . Then the dualizing bundle V* := V v <8> &g/p is given by 
V* — G x P {V y ® C 2p ) as a homogeneous vector bundle. The pull-back 
of V* — v G/P to % via (|2.2|) defines the trivial vector bundle n R x V v — > %, 
and induces an injective morphism i* : C°°(G/P, V*) — > C°°(n R )(8)V*. The in- 
finitesimal representation of the regular representation C°°(G/P, V*) is given 
as an operator on C°°(%) <g> V w by 

(2.3) dir* x {C) := (d\ v + 2pid v v,a{C, •)) - 0(C, •) <g> idyv . 

We may regard dK* x (C) as an End(K v )-valued holomorphic differential oper- 
ator on u~ = % ®r C. 

Let P be a vector space over C. We recall that the Weyl algebra T>(E) 
is the ring of holomorphic differential operators on E with polynomial coef- 
ficients. 

Definition 2.1. We define the algebraic Fourier transform as an isomor- 
phism of two Weyl algebras on E and its dual space P v : 

V{E) P(P V ), T^f, 

induced by 

IT ^ d 

— :=-(•, z- := — , l<j<n = dimP, 
dzj <Kj 

where (zi, . . . , z n ) are coordinates on E and (d, . . . , £ n ) are coordinates on 
P v . The definition does not depend on the choice of coordinates. 

In what follows, we shall regard distributions as generalized functions a 
la Gelfand rather than continuous linear forms on C%°(X). 
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Remark 2.2. Suppose E R is a real form of E. We define 'Fourier transforms' 
T c : S'(E R ) -> 0(F V ), / H- / f{x)e^dx, 



where £'{E^) stands for the space of compactly supported distributions. We 
note 

(2.4) (7k/)(0 = (7- c /)(-z0- 

In |H| ^5f we have adopted T c instead of J-r. An advantage of T c is that 



the algebraic Fourier transform defined in Definition [O] satisfies 
(2.5) f = 7 c oT oj" 1 for all T £ V(E). 



See also Remark |2.4| below. 

Suppose P is a G"-compatible parabolic subgroup, and P' :— P H G' — 
Z/expfn^)'. We define a subspace of Hom(V, iy)-valued tempered distribu- 
tions on nj^ by 

Sol(V, W) A := {F G (8) Hom(V, W) : F satisfies (|2j) and 

where 

(2.6) o F(Ad(/- x )-) o A(/- x ) = F(-) for all / G L', 



(2.7) (o?7r^(C) <g> id w + idyv ®dv{C))F = for all C G (ri 



Combining p5[ and ||26|| , we obtain: 



Theorem 2.3 (F-method for continuous operators). Let G be a reductive 
linear Lie group, and G' a reductive subgroup. Suppose P is a G' -compatible 
parabolic subgroup of G, and P' = P fl G' . We assume 

(2.8) P'N-P = G. 

Then the Fourier transform J-jg of the distribution kernel induces the follow- 
ing bijection: 

(2.9) Hom G ,(C°°(G/P, V), C°°{G'/P', W)) ^ Sol{V, W) A . 
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Remark 2.4. Theorem |2.3| extends the following bijection: 

(2.10) DiR G .{C°°(X, V), C°°(F, W)) ^ Sol{V, W) A n Pol(n H 



which was proved in (cf. P5| ) without the assumption (ETSj) . In the case 



where n + is abelian, the bijection ( |2.10|) is compatible with the symbol map 



Diff G ,(C°°(X, V), C°°{Y, W)) -> Pol(n+) <g> Hom c (V r , W), 



if we use instead of Jtr (see [1X81 B5 



3 Conformally equivariant restriction opera- 
tors 

In this section we set up some notation for conformally equivariant operators 
in the setting where G' C G act conformally on two pseudo-Riemannian 
manifolds Y G X, respectively. 

Let X be a smooth manifold equipped with a pseudo-Riemannian struc- 
ture g. Suppose a group G acts conformally on X. The action will be denoted 
by Lh '■ X — > X, x i— > LhX for h E G. Then there exists a positive- valued 
function Q on G x X such that 

L* h (gL h x) = &(h,x) 2 g x for any h E G, and x E X. 

Fix A E C, and we define a linear map zuxih^ 1 ) : C°°(X) -> C°°(X) by 

(^(O/Xz) :=n(h,x) x f(L h x). 

Since the conformal factor f2 satisfies the cocycle condition: 

Q(hih 2 , x) = Q(hi, L h2 x)Q(h 2 , x) for hi,h 2 E G,x E X, 

we have formed a family of representations vo\ = to* of G on C°°(X) with 



complex parameter A (see pi], Part I] for details). 

Remark 3.1. 1) If G acts on X as isometries, then Q = 1 and therefore the 
representation w\ does not depend on A. 

2) Let n be the dimension of X. Then in our normalization, (zcr n ,C°°(X)) 
is isomorphic to the representation on C°°(X, Qx) where fix denotes the 
bundle of densities. 
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Let Conf(X) be the full group of conformal transformations on (X,g). 
Suppose Y is a submanifold of X such that the restriction g\y is non- 
degenerate. Clearly, this assumption is automatically satisfied if (X, g) is 
a Riemannian manifold. We define a subgroup of Conf(X) by 

Conf (X; Y) = {y? G Conf (X) : <p(Y) C Y}. 

Then if induces a conformal transformation on (Y,g\y), and we get a 
natural group homomorphism 

Conf(X;F) ->■ Conf(Y). 

Thus, for A, v G C, we have the following two representations: 

vol : Conf (X) GLcCC 00 ^)), 
< : Conf(X; Y) ->• GL c (C°°(y)). 

We are ready to state the following problem: 

Problem 3.2. 1) Classify (A, v) G C 2 such that there exists a non-zero 
continuous/differential operator 

T Ku : C°°(X) -> 

satisfying 

^(/i) o T v = T v o rof (/i) for all /i G Conf(X; K). 

2) Find explicit formulas of the operators T\ iV . 

We begin with an obvious example. 

Example 3.3. Suppose A = v. We take T\ tV to be the restriction of functions 
from X to Y. Clearly, T\ >u intertwines w\ and w v . 



Problem |3.2| concerns with a geometric aspect of the general branching 
problem for representations with respect to the restriction G 4- G' in the case 
where 

(G, G') = (Conf (X), Conf Y (X)). 

We shall see in Section [| that a special case of Problem |3]2| is a special case 
of Project A. In Section |], we shall write and for vj\ = vuf and 
ro^ = , respectively, in order to emphasize the groups G and G". . 
We continue some further examples of Problem [3.2|. 
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Example 3.4. Suppose X = Y. Then G = G' . Let n be the dimension of 
the manifold, and we consider the following specific parameter: 

1 1 

A = —n — 1, v = -n + 1. 

2 ' 2 

Then the Yamabe operator Ax satisfies w v o Ax = Ax ° vo\, where 

- — n — 2 

A x := A x - —, ttK. 

4(n — 1) 

Here Ax is the Laplacian for the pseudo-Riemannian manifold (X,g), and k 
is the scalar curvature. 

In particular, if X is the direct product of two spheres S p x S q endowed 
with the pseudo-Riemannian structure g$p © ( — <7s«) °f signature (p, g), then 
the kernel Ker(Ax) gives rise to an important irreducible unitary represen- 
tation of Conf(X) ~ 0(p + l,q+ 1), so-called the minimal representation for 
p + q even, p,q > 1, and p + q > 6 |T3], ^TJ in the sense that its annihilator in 
the enveloping algebra U(o(p+q+2, C)) is the Joseph ideal @, ^7[| . The same 



representation is known to have different realizations and constructions, e.g. 



the local theta correspondence fL0 |, and the Schrodinger model 

Example 3.5 (Juhl |IT[|). Let X be the standard sphere S n and Y = S 71 ' 1 
a totally geodesic hypersurface ('great circle' when n = 2). Then we have 
covering maps 

0(n + 1, 1) ^Conf(X), 

u u 

0(n,l) ^Confy(X). 

We set 

(G,G') = (0(n + l,l),0(n,l)). 

Then non-zero G'-equivariant differential operators T\ iU : C°°(X) — )• C°°(y) 
exist if and only if the parameter (A, v) satisfies v — A £ {0, 2, 4, • • • }. In this 
case dimDiff G /(C oo (X),C 0O (y)) = 1. 

In order to describe this differential operator T\ jU explicitly, we use the 
stereographic projection 



(3.1) 5 n ^M n U{oo}, (s,VT^cu) h- \j- — -co, 

1 — r S 
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F(ru) := (1 + r 2 )- x f(- -, -co) for r > and u E S n ~\ 



and the corresponding twisted pull-back for the conformal map ( |3.1| ) (see [21 
Part I]) 

(3.2) l* x : C°°{S n ) ^ C°°(R n ), f^F 

is given by 

. 1 — r 2 2r 
1 + r 2 ' 1 + r 2 

In the coordinates, the submanifold Y corresponds to the hyperplane x n = 
via (|3.1|) . Then for u — A = 21 (I £ N), the equivariant differential operator 
T\ yU is a scalar multiple of the following differential operator: 

C x>v : C°°(W) C^iW 1 - 1 ), 

. . -^—K 2 2l ~ 2 i l -fi,\ + v-n-l \ A , / d \ 2Z-2j 

< 3 - 3 > /-g^r^n( — 2 — 'i*- 

The differential operator C\ ;U can be written by using the Gegenbauer poly- 
nomial as follows. Let C^(t) be the Gegenbauer polynomial of degree N, 
and we inflate C^(t) to a polynomial of two variables by 
(3.4) 

j — ? — 1 

Then 



A. Juhl [LT discovered the formula ( |3.3p by using recurrence relations. In 
23[ we have given a new proof by using a differential equation according to 
the F-method. Theorem |4]^ gives yet another new proof of the formula ( |3.3|) 
based on the residue calculations of continuous intertwining operators with 
meromorphic parameter. 



4 Conformally equivariant non-local opera- 
tors 

In this section we analyze both non-local and local, conformally equivariant 
operators in the setting of Example [3.5| by using the F-method. 
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From the viewpoint of representation theory, Example |3.5| deals with dif- 
ferential intertwining operators between spherical principal series represen- 
tations and those when (G, G') = (0(n+ 1, 1), 0(n, 1)). In this case 
all the assumptions for Theorems [Ll] and [O] are fulfilled, and therefore we 
tell a priori that both of the sides in ( |1.2|) are finite dimensional. In the joint 



work |£6| with B. Speh, we prove the full classification of such intertwining 
operators between line bundles for both non-local and local ones with ex- 
plicit generators, which gives the first complete example of Project A. Since 



most of the proof in pSf does not use the F-method, it would be of help for 
further understandings to illuminate some of the key results of from the 
scope of the F- method (Section [|). This is the goal of the current section. 
In particular, we use systematically the F-method and explain an idea of 
how to discover functional equations among non-local intertwining operators 
(Theorem fO)| ) and how the local operators Cx, v appear as the residues of 
non-local operators Ax, v with meromorphic parameter (A, v) e C 2 (Theorem 



ID- 

We set |x| := (./• 
(A, v) E C 2 satisfies 



i + 



+ Xn-i) 2 for X 



n—1 1 



on— 1 



If 



(4.i; 

then 



the integral operator 



Re(A - v) > and Re(A + v) > n - 1, 
: = |£n| A+ " _n (M 2 + x n)~ u ls locally integrable on IR n , and 



-^C c 



on— 1^ 



/ >->■ / f{y, y n )K(x - y, -y n )dydy n 



extends to a G"-intertwining operator via (|3.2|) 

Ax, v : C°°(S n ) C°°(S n - 1 ), 



namely, 



A x ,v ° zu\(h) = m„\h) o A\,u for all G G' . 



,G' 



The important property of our intertwining operator Ax,v is the existence 
of the meromorphic continuation to (A, v) 6 C 2 (see Theorem [4.1| ) and the 
functional equations satisfied by Ax, u and the Knapp-Stein intertwining op- 
erators (see Theorem |4.5|) . We note that the celebrated theorem 0, 3 on 
meromorphic continuation of distributions does not apply immediately to 
our distribution K£ v because the two singularities x n = and \x\ 2 + x 2 = 
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(i.e. the origin) are highly degenerate and not transversal. Further, it is more 
involved to find the location of the poles and zeros. In we have found all 
the poles and residues explicitly, and in particular, we have established the 
following theorems: 

Theorem 4.1. We normalize A x , u ■= r( A+,-n+i )r( -A^7y^A^- Then A x ,u ■ 

C°°(S n ) — > C°°(5' n_1 ) is a continuous operator that depends holomorphically 
on (A, v) G C 2 and satisfies 

A\,v o zuf(h) = wf(h) o A x ,v for all h G G' . 

Theorem 4.2. If v - \ = 2l with I G N then 

' u 2 2 'I» x ' u ' 



The proof of Theorem [4.2| in [EH] is to use explicit formulae of the action of 



A\ yU and C\ :l/ on i^-fixed vectors. Instead, we apply here the F-method and 
give an alternative proof of Theorem |4.2| , which is of more analytic nature 
and without using computations for specific K-types. 

We write K^ v and K% v for the distribution kernels of A\ tU and C\ iV , 
respectively. For the open domain ( f4.1|) , we have 



K\,v( x i x n) — p^ A+ „_ n+1 ^p^_ y ^ \ x n\ + + X n ) . 

Proposition 4.3. 1) The tempered distribution J-^K^ V G S'(M. n ) is a real 
analytic function (in particular, locally integrable) in the open subset {((, ( n ) G 
R n_1 : |e| > |Cn|}; where \(\ = ■ - + Cn-i) 5 - analytic continuation 

(cf. we have for \(\>\( n \ 

(4.2) 

F( X -» A + i/+l-n l. C* 
Wc^a,WvS)W T(v)2 u ~ x 2 ' 2 '2' |C| 

2) Suppose v — X — 21 (I en). Then 

~ - — ' \ n — 1 
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Proof. 1) We use the integration formula 

* *L / |t| A -^^-, +f (|tei)e-^rft, 

r(v)\z\- v+ s- Jr 

where K^it) denotes the if-Bessel function. We then use the following inte- 
gration formula: 

i 

x y K lx (ax) cos(bx)dx 

= 27-1.-7-1 ^ + 7 + 1 )r( 1 + 7 - H ) F f + 7 + 1 1+7^ 1 _ &\ 
v 2 ; v 2 ^ u 2 ' 2 '2 a 2 

for Re(-7 ± //) < 1, Re a > 0, 6 > 0, see |g 6.699.4]. 

2) Clear from definition. □ 



o 



Proof of Theorem By using the following formula of the Gegenbauer 
polynomial of even degree 

c 2/l x J /!r(/i) 2 ^H _Z ' z + A*' 2' 

we get 



-p K A _ V V X" K C 



for |C| > la- 
in view that Suppi^ C {0} for v - A G 2N (||), both and 

JF C K^ V are holomorphic functions on C n . Hence Theorem [4.2| follows. □ 



Remark 4.4. The assumption (|2.8|) of Theorem ^73] is satisfied, and therefore, 



Hom G / (C 00 (G/P,Cx),C 00 (G'/P',C v )) can be identified with 

Sol(X, u) h = {F G 5'(lR n ) : F satisfies the following three equations}. 

F(m-) = F(-) for m G 0(n - 1) x 0(1), 
(EO|" + A-^))F = 0, 

(^-^A r gF = (l<j<n-l). 
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The differential operators in the last equation are the fundamental differential 
operators on the light cone [^0[ Chapter 2] (or Bessel operators in the context 
of Jordan algebras). The heart of the F-method is that this differential 
equation explains why the hypergeometric functions (and the Gegenbauer 
polynomials as special cases) arise in the formula of T c K Xv and F c Kx „ in 
Proposition fD| 

We recall that the Riesz potential 



Kn-\,\( X > X n) '■— — ( X l + ' ' ' + X n) X 



gives the normalized Knapp-Stein intertwining operator by 

T n -x,x '■ C°°(W l ) ->■ C°°(R n ), f^K^_ xx *f. 

Then T n ~x,x depends holomorphically on A 6 C and satisfies 

w x (h) o tn-\,x = t n -x,x o w°_ x {h) for all heG. 

Here are the functional equations among the three operators: our operators 
A\„, the Knapp-Stein operators l~ v ^ m - v for G' and %,-xx for G. 



Theorem 4.5 (p6fl). Let m — n — 1. 

m 

~ ~ 71 2 ~ 

(4-3) Ty^m—v ° r(z> / ) ^^ ,m— V ' 

n 

(4.4) Ax,u O f n -X,X = yj^Au-Xm- 

Heuristic idea of proof based on the F -method. Combining (|4.2p and 



e f (2A-») 



7T 2 



(4.5) j- c (^_ AiA )(c,Cn) = 22A _ nr(A) (icr+c 



2 



we would have the following identity of holomorphic functions on {(£, £ n ) G 
C ra : |C| > |Cn|} a s analytic continuation: 



x) 



r(A)r(z/)2 A +^-« " 2 ^ iV 2 ' 2 '2' |C| 2 ' 
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Then (O) would be reduced to Kummer's relation: 



F(a, (3; 7; z) = (1 - ^"^F( 7 - a, 7 - # 7! 4 
The identity (J4.4[) is similar and simpler. □ 



The advantage of the F-method indicated as above is that we can discover 
such identities as (|4.3|) and (|4.4j) in a natural way. This direction will be 
discussed also in ^3] for functional identities of differential operators (e.g. 
factorization identities in |ll|]). In the case of non-local operators, we need a 
careful treatment of distributions. A rigorous and completely different proof 



of Theorem 4.5 and its variants is given in [26| 



References 

[1] M. Atiyah, Resolution of singularities and division of distributions, 
Comm. Pure Appl. Math., 23, (1970) 145-150. 

[2] I. N. Bernstein, S. I. Gelfand, Meromorphic property of the functions 
P x , Funktsional Anal, i Prilozhen., 3, (1969) 84-85. 

[3] B. Binegar, R. Zierau, Unitarization of a singular representation of 
SO(p,q), Comm. Math. Phys., 138, (1991) 245-258. 

[4] P. Clare, On the degenerate principal series of complex symplectic 
groups, J. Funct. Anal. 262, (2012) 4160-4180. 

[5] J.-L. Clerc, T. Kobayashi, B. 0rsted, and M. Pevzner, Generalized 
Bernstein-Reznikov integrals, Math. Ann., 349, (2011) [395-431 . 

[6] H. Cohen, Sums involving the values at negative integers of L- functions 
of quadratic characters, Math. Ann., 217, (1975) 271-285. 

[7] P. B. Cohen, Y. Manin, D. Zagier, Automorphic pseudodifferential op- 
erators, Progr. Nonlinear Differential Equations Appl., 26, Birkhauser, 
1997 pp. 17-47. 

[8] I. S. Gradshteyn, I. M. Ryzhik, Table of Integrals, Series, and Products, 
Seventh edition. Elsevier, 2007. xlviii+1171 pp. 



18 



[9] M. Harris, H. P. Jakobsen, Singular holomorphic representations and 
singular modular forms, Math. Ann., 259, (1982) 227-244. 

[10] J.-S. Huang, C.-B. Zhu, On certain small representations of indefinite 
orthogonal groups, Represent. Theory, 1, (1997) 190-206. 

[11] A. Juhl, Families of conformally covariant differential operators, in- 
curvature and holography. Progr. Math., |275| . Birkhauser, 2009. 

[12] M. Kashiwara, T. Oshima, Systems of differential equations with regular 
singularities and their boundary value problems, Ann. of Math., 106, 
(1977) 145-200. 

[13] T. Kobayashi, Discrete decomposability of the restriction of A q (\) with 
respect to reductive subgroups and its applications, Invent. Math., 117, 
(1994) 1181-2051; Part II, Ann. of Math., 147, (1998) |709-729| ; Part III, 
Invent. Math., 131, (1998) |229 256 . 

[14] T. Kobayashi, Conformal geometry and global solutions to the Yamabe 
equations on classical pseudo-Riemannian manifolds, Proceedings of the 
22nd Winter School "Geometry and Physics" (Srm, 2002), Rend. Circ. 
Mat. Palermo (2) Suppl. 71, 2003, pp. |I5~40] 

[15] T. Kobayashi, Multiplicity-free theorems of the restrictions of unitary 
highest weight modules with respect to reductive symmetric pairs, pp. 
45-109, Progr. Math., |255|, Birkhauser, Boston, 2008. 

[16] T. Kobayashi, Branching problems of Zuckerman derived functor mod- 
ules, Contemp. Math., 557, pp. 23-40. Amer. Math. Soc, 2011. (avail- 
able at |arXiv: 1104.43991) 

[17] T. Kobayashi, Restrictions of generalized Verma modules to symmetric 
pairs, Transform. Group, 17, (2012) |523 546 . 



[18] T. Kobayashi, F -method for constructing equivariant differential oper- 
ators, to appear in Contemp. Math. Amer. Math. Soc, (available at 
larXiv: 1212.6861 )• 

[19] T. Kobayashi, Propagation of multiplicity-freeness property for holo- 
morphic vector bundles, Progr. Math. 306, Birkhauser, 2013 (in press), 
(available at |arXiv:0607004| ). 



19 



T. Kobayashi, G. Mano, The Schrodinger model for the minimal repre- 
sentation of the indefinite orthogonal group 0(p, q), Mem. Amer. Math. 
Soc. (2011), 212, |no. 10001 , vi+132 pp. 



T. Kobayashi, B. 0rsted, Analysis on the minimal representation of 
0(p,q). Part I, Adv. Math., 180, (2003) |486-512j ; Part II, ibid, |513~| 
"5501 Part III, ibid, |551-595j . 



T. Kobayashi, B. 0rsted, M. Pevzner, Geometric analysis on small uni 
tary representations of GL(n,18L), J. Funct. Anal., 260, (2011) |1682- 
T720| . 



T. Kobayashi, B. 0rsted, P. Somberg, V. Soucek, Branching laws for 
Verma modules and applications in parabolic geometry, Part I, Part II, 
in preparation. 

T. Kobayashi, T. Oshima, Finite multiplicity theorems for induction and 
restriction, preprint, 32 pp. |arXiv:1108.3477| 

T. Kobayashi, M. Pevzner, Rankin-Cohen operators for symmetric 
pairs, preprint, 53pp. |arXiv 1301.2111 . 



T. Kobayashi, B. Speh, Some intertwining operators for rank one or- 
thogonal groups, in preparation. 

B. Kostant, The vanishing of scalar curvature and the minimal repre- 
sentation of 50(4,4), Progr. Math., 92, Birkhauser, 1990, pp. 85-124. 

R. A. Rankin, The construction of automorphic forms from the deriva- 
tives of a given form, J. Indian Math. Soc, 20, (1956) 103-116. 

B. Sun, C.-B. Zhu, Multiplicity one theorems: the Archimedean case, 
Ann. of Math., 175, (2012) 23-44. 



20 



